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This invention is best described in the following statement: 



METHOD AND APPARATUS FOR MOTION ESTIMATION 
Technical Field of the Invention 

The present invention relates generally to image processing and, in particular, to 
a method and apparatus for estimating motion. 
5 Background Art 

Known arrangements for performing motion estimation of video images 
typically use one of a number of methods. A first method involves estimation of 
illumination partial derivatives to calculate the optical flow. Another method involves the 
local (block based) calculation of the cross-correlation between successive images and 
10 using the cross-correlation maxima to estimate the shift between images. A third method 
utilizes estimation of the motion blur in any one image (using a variety of methods such 
as blind de-convolution, maximum likelihood estimation etc) to then use as a motion 
vector in optical flow. 

The above arrangements suffer each from the disadvantage of being time 
15 consuming and not necessarily very robust or reliable, particularly for large magnitudes of 
motion. 

Motion estimation is a useful tool in many aspects of image processing. Motion 
estimation is used in video compression (for example MPEG2). Motion estimation is also 
useful in identifying those portions of an image that may require updating on a display. 

20 Motion estimation also permits various portions of an image may have assigned to them, 
for example as metadata, motion vectors that are interpretable to provide for enhanced 
image processing. Motion detection may also be used for object tracking and content 
analysis and many other computer vision applications. 

One deficiency of existing motion estimation arrangements is that they operate 

25 between adjacent frames of an image sequence, or between odd and even fields of an 
interlaced image within such a sequence. As a consequence, known arrangements are 
limited to the image sampling rate either as frames or fields. However, motion may yet 
occur during the sampling of, or between, such frames or fields. Known arrangements 
also lack fine resolution of the magnitude of motion. For high quality reproduction, 

30 which includes consideration of issues such as storage efficiency, such motion is desired 
to be considered. 

Disclosure of the Invention 

It is an object of the present invention to substantially overcome, or at least 
ameliorate, one or more disadvantages of existing arrangements. 
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According to a first aspect of the invention, there is provided a method for 
estimating motion of a moving object, said method comprising the steps of: 

capturing at least first and second blurred images of said moving object using a 
capture device having a predetermined field rate, the blur in said images arising from at 
least motion blur of said object, said images being captured within a window period, the 
window period being shorter than a period defined by a field rate of the capture device; 

generating an error function, said error function being a function of said first 
blurred image and said second blurred image; 

minimising said error function; and 

estimating said motion of said object from said minimised error function. 

According to an other aspect of the invention, there is provided a method for 
estimating motion of a moving object, said method comprising the steps of: 

capturing at least first and second blurred images of said moving object using a 
capture device having a predetermined field rate, the blur in said images arising from at 
least motion blur of said object, said images being captured within a window period, the 
window period being shorter than a period defined by a field rate of the capture device; 

generating an error function, said error function comprising a cross-correlation 
term being a cross-correlation between said first blurred image and said second blurred 
image; 

minimising said generated error function; and 

estimating said object motion from said minimised error function. 

According to another aspect of the invention, there is provided an apparatus for 
implementing any one of the aforementioned methods. 

According to another aspect of the invention there is provided a computer 
program product including a computer readable medium having recorded thereon a 
computer program for implementing any one of the methods described above. 

According to another aspect of the invention there is provided a motion 
estimation system comprising: 

a capture device for capturing at least first and second blurred images of a 
moving object, the blur in said images arising from at least motion blur of said object, 
said images being captured within a window period, the window period being shorter than 
a period defined by a field rate of the capture device; 
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means for generating an error function, said error function comprising a cross- 
correlation term being a cross-correlation between said first blurred image and said 
second blurred image; 

means for minimising said generated error function; and 
5 means for estimating said object motion from said minimised error function. 

Brief Description of the Drawings 
A number of preferred embodiments of the present invention will now be 
described with reference to the drawings, in which: 

Fig. 1 shows blur functions b x and b 2 representing lineal (motion) blur of spatial 
10 extent 2a x and 2a 2 respectively occurring in a direction at angle 6 to the x axis; 

Figs. 2A and 2B illustrate the x r components of the respective blur functions bj 
and bi resulting from two sequential, non-overlapping, uniform exposures with equal 
duration; 

Figs. 3A and 3B illustrate the x f components of the respective blur functions bj 
15 and b2 resulting from two uniform exposures centred on the same time, but with the one 
exposure twice the duration of the other; 

Figs. 4A and 4B illustrate the x component of the respective blur functions bi 
and 62 , resulting from two uniform exposures with the same start times, but with the one 
exposure twice the duration of the other; 
20 Figs. 5 A and 5B illustrate the x component of the respective blur functions bj 

and 2b2(x)+bj(x), resulting from a linear combination of bi and bj\ 

Figs. 6 A and 6B illustrate the x' components of the respective blur functions bj 
and 62 resulting from two uniform exposures centred on the same time, but with the one 
exposure three times the duration of the other; 
25 Figs. 7A and 7B illustrate yet another embodiment wherein the x f components 

of the respective blur functions b\ and 62 resulting from two non-uniform exposures 
centred on the same time, but with the one exposure twice the duration of the other; 

Fig. 8 is a schematic block diagram of a general purpose computer upon which 
embodiments of the present invention can be practiced; 
30 Fig. 9 is a schematic block diagram of an apparatus for estimating blur 

parameters of an image; 

Fig. 10 is a schematic block diagram of a structure of an active image sensor 
used to capture blurred images g\ and g2 of Fig. 9; 
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Fig. 1 1 is a timing sequence to be used with the sensor of Fig. 10 to capture the 
two overlapping blurred images g\ and g2 with the exposure pattern of Figs. 4A and 4B; 
and 

Figs. 12A to 12E provide comparisons between image capture according to prior 
5 art arrangements and those according to embodiments of the present invention. 

Detailed Description including Best Mode 
The embodiments of the invention described herein utilise the special 
exposure/electronic shuttering properties of a novel active image sensor in conjunction 
with an error minimisation scheme tailored to a sequence of images related by a specific 
10 blur relationship. In general, this method uses a combination of pure (symmetric) blur 
combined with pure (asymmetric) shift to obtain a maximum likelihood estimate of the 
underlying motion causing the blur and shift. 

Consider an ideal two dimensional digital image signal, represented as a function 
f(x 9 y), where x and y are the horizontal and vertical image coordinates. The image 
15 signal f(x,y) is defined for all integer values of x and jy, and undefined for all non-integer 
values of x and jy. The digital image signal f(x,y) is typically obtained by sampling an 
analog image, for instance, an image on film. 

In practice, this ideal image signal f(x,y) is distorted by various image 
degradations. In this specification image blurring is considered as well as additive noise. 
20 The image blurring to be considered is a special form of blur, namely lineal blur typically 
caused by motion in a plane at an oblique angle to a line of sight of a camera used for 
capturing the image. The blur is the projection of the object's motion into the two 
dimensional planeof the camera or image sensor. It is assumed that motion is 
approximately linear over a small period. The blurred image is represented as g v In the 
25 preferred embodiment of the invention, two different blurred images are used, g l and g2- 
The blurred images g x and g 2 have a special relationship, examples of which are 
described in the embodiments. Such related images g ] and g2 can typically be derived 
using the multiple readout options that are realised with the active sensor to be described. 

The blurred images g x and g 2 can be represented as the result of a pair of blur 
30 functions b } and b 2 performed on the image signal f(x,y), assuming the blur is uniform 
over the whole image. Therefore, the blurred images g x and g 2 are represented using the 
convolution (+) operator as: 
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gi(x,y)=f(x,y) *b,(x,y) + n,(x,y) 



(1) 



g2(x,y)=f(x,y) * b 2 (x,y) + n 2 (x,y) (2) 

5 where n\ and «2 are additive noise signals. 

An assumption is made that the motion is linear and uniform over the image 
region under consideration. The blur functions b x and b 2 represent lineal (motion) blur of 
spatial extent 2a x and 2a 2 respectively occurring in a direction at angle Oto the x axis is 
illustrated in Fig. 1. Consequently the blur functions b x and b 2 can be represented as 
10 follows: 

b l (x,y) = ^™*(-£-)W) (3) 

2a, 2a x 



b 2 {*,y) = ^reet{£^W-yl) (4) 
2a 2 2a 2 



wherein 

15 red 



K 2sj 



= 0 for p>s 

1 for \p\<s ,and (5) 

8(p) is the Dirac delta function (6) 
20 and (x\ y f ) are rotated coordinates having the following relation: 



x' = +xcos(^) + j/ sin(0)] 
y f = -xsin(d) + y cos(0)\ 



(7) 



As will be apparent from Fig. 1, the rotated coordinates have one coordinate 
25 aligned with the direction of blur. The centre of the second blur function b 2 is offset by 
an amount (x 0 ,y 0 ) from the first blur function b r In essence the blur functions b x and b 2 
convert all point intensities in the image signal f(x 9 y) into linear smears of length 
directly related to the exposure time and the image velocity. 
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The two dimensional Fourier transform G(u,y) of an arbitrary signal g(x,y) is 
defined as: 

<X> CO 

G(u, v) = J §g(x,y)exp(-2m[wc + vy\dxdy (8) 

— oo— oo 

5 where u and v are the spatial frequency coordinates. Consequently, it is possible to write 
the transforms G; and G2 of the blurred images gj and g2 as: 

G I (u,v)=F(u,v).B I (u,v) + Nj(u,v) (9) 

10 G 2 (u, v) =F(u, v). B 2 (u, v) +N 2 (u, v) (10) 

The discrete Fourier transforms Bj and B 2 of the blur functions b } and b 2 have a 
simple (one dimensional) form when expressed in relation to rotated frequency axes, 
namely: 

u' = +u cos(0) + vsin(#)l 
15 > (11) 

v f = -w sin(0) + vcos(0) J 



^i(^v>= ; , 17 (12) 

z^w a, 



5 2 (w,v) = ; , 2 ' .exp(-27a[ M '*;]) (13) 

2m/ <2 2 



20 



From Equations (9), (10), (12) and (13), the transforms Gj and G 2 can be 
rewritten as: 

^ , x t^s x sin(2m<'a 1 ) 

G,(w,v) = F(u 9 v). V, (14) 



25 G 2 (», v) = F(u, v). sin ( 2 ^' fl ^ . exp(-2^[K^ ]) + N (15) 

2ttw a 2 



For simplification, the noise spectra term Nj and N 2 in Equations (9) and (10) has 
been substituted by a single noise spectrum term N in Equation (15). From the two 
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blurred images gj and g 2t two equations and four "unknowns" (viz the function F(u,v), 
as well as the parameters a v a 2 , and x ( 0 ) are derived. By constraining the parameters a,, 
a 2 , and x' 0 , the unknowns can be reduced to two, and a unique solution thus exists for the 
simultaneous Equations (14) and (15). 
5 The above deduction can be performed upon two or more images. Embodiments 

using two images include: 

(a) sequential, non-overlapping, uniform exposures with equal duration; 

(b) both uniform exposures centred on the same time but one exposure twice 
the duration of the other; 

10 (c) overlapping, uniform exposures with the same start time and one 

duration twice the other; 

(d) variations of (b) and (c) with integer exposure multiples greater than two; 

and 

(e) variations of (b) and (c) with non-uniform exposures. In this case the 
15 exposures are preferably representable by multiple convolutions of a uniform exposure 

pattern. 

Case (a) above, illustrated in Figs. 2A and 2B, will not be considered further 
because it reduces to a well known solution of estimating pure time delay (ie. pure 
translational shift on the image). The blur parameter is not used in this solution. This 
20 solution only utilises the elapsed time or spatial separation between the two uniform 
exposures with equal duration. 

Figs 3A and 3B illustrate the x f components of the respective blur functions b\ and 
62 resulting from two uniform exposures centred on the same time, but with the one 
exposure twice the duration of the other, thereby representing Case (b) above. This 
25 particular relationship between the two exposures results in the following parameters: 

a x ~ a 

. \ ( 16 ) 

y'o=o ^ 

Applying these parameters to Equations (14) and (15) gives the Fourier spectra: 

30 
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G,{u,v) = F(w,v).— — ; (17) 

2nu a 



sm(47ru'a ) 
Atvucl 



G 2 (u, v) = F(u , v). ;_ r + (w, v) (18) 



5 Using a trigonometric expansion, a relationship between Gj and G 2 can be found as: 

2sin(2/zu'a )cos(2mi'a) 

G 2 (u,v) = F(u,v). i —L t - + N(u,v) (19) 

4mi a 



G 2 (u,v) = G,(w,v).cos(2;zw'a) + N(u 9 v) (20) 

10 

The difference between blurred images gj and g 2 can be measured using total 
modulus-squared spectral "deviation" t, integrated over a representative region of the 
spatial frequency. The modulus-squared spectral "deviation" t can be written as a 
function of the blur parameter a : 

15 

t( a )= jjjN(u 9 v)\ 2 dudv= j[\G 2 (u 9 v) - G, (w, v).cos(27zw'a)| 2 dudv . (21) 



The blur estimation problem can now be expressed in terms of a least-squares 
estimation, in which the image "discrepancy" t(a) is minimised with respect to the blur 
20 parameter a . The stationary condition at a minimum is simply 



f- = 0. (22) 

da 



25 



However, Equation (21) can be further expanded as: 

t( a )= H\g 2 \ 2 +|G 1 | 2 cos 2 (27rw'a)-cos(27TM^)(G 1 G + 2-fG + iG 2 )}/w^v. (23) 
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-(1 + cos(Wa)) - cQs(2nu'a){G x G*i + G\G 2 )\dudv . (24) 



The blurred images g, and g 2 are real even functions, resulting in the Fourier spectra 
thereof Gj and G2 having Hermitian symmetry. Hence, the following apply: 

5 

|G 2 | = real, even 

\G X J = real, even ► (25) 

(g,G* 2 + G\G 2 )= real, even 



Equation (24) can be expressed in terms of a complex exponential, because the 
imaginary sine components identically sum to zero when the symmetry conditions of 
10 Equation (25) apply. 



Applying the coordinate transforms of Equation (11) the transformation can be 
15 rewritten as: 

u'a = ua cos 6 + va sin 0 (27) 

The image "discrepancy" / in Equation (26) is a function of two variables, and 
therefore t(a, 9) must be minimised with respect to both parameters, a and 9. Since a and 
20 0have a polar structure, such allows the introduction of two new parameters X and Y with 
a rectilinear form as follows: 

acos9 = X) 

(28) 

asin0 = 7j 

Applying Equation (28) to Equation (26) results in the following: 

25 

t(a,9) = e(X,Y) (29) 
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(30) 



The first term in Equation (30) is a constant term related to the total "energy" in 
the two signals gj and g2, which may be ignored in order to minimise the image 
"discrepancy" The second term is the (two-dimensional) auto-correlation of blurred 
image g x which has been scaled by a factor of one half, while it's coordinates have been 
scaled by two. The third term is the cross-correlation of the blurred images g x and g 2 plus 
the corresponding 180 degrees rotated form. 

Therefore, in the preferred embodiment, the image "discrepancy" t function can 
be minimised with respect to X and Y by calculating the image "discrepancy" / for all 
values of X and Y using the Fast Fourier Transform, which is the most efficient way to 
calculate large kernel correlations. If only a small blur is expected, then the correlation 
calculations can be limited to a small zone, and take advantage of computational 
efficiencies available. Additional efficiencies are possible using the symmetries apparent 
in Equation (30). 

In another embodiment, the exposures or blur functions are chosen to have the 
same start times, and blur function bi has twice the duration of blur function bj. Figs. 4A 
and 4B illustrate the x' component of the respective blur functions bi and 62. This 
particular relationship between the two exposures results in the following parameters: 



It is noted that these exposures or blur functions bj and 62 can be synthesised 
from Case (a) discussed above. The first exposure is just bj, whereas the first and second 
exposures of Figs. 2 A and 2B can be averaged to synthesise b 2 as shown in Fig. 4B. 



a, = a 




> 



(31) 
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Applying the parameters of Equations (31) to Equations (14) and (15) gives the 
Fourier spectra: 

G>,v) - F(m;v). ; , } (32) 
2;rw a 



sin(4/rw'a ) * , 

G 2 (u,v) = F(u 9 v). — -.exp(-2mu'a)+N(u 9 v) (33) 

4m' a 



From Equations (32) and (33), the noise term N(u,v) may be expressed as 

follows: 

N(u,v) = G 2 (w 3 v)-Gi(w,v).cos(2^w'fl).exp(-2^zw'a) (34) 

10 

Again, the modulus-squared error or "deviation" / can be written as a function of 
the blur parameter a : 



t( a )= ^\N(u,v)\ 2 dudv = J^|G 2 («,v)-G 1 (w 5 v)xos(2^'a).exp(-2^w / fl)| 2 rfwrfv. (35) 
However, Equation (35) can be further expanded as:. 
/(a)= J[|G 2 | 2 +|G 1 | 2 cos 2 (2^w^)}/Mrfv 



15 



rr/ \ ' (36) 

- \\\plG l .exp(--2mu t a J+ G*G 2 ,exp(2mu'a)).cos(27iu'a)dudv 



|G," 



1 2 

20 + 2^' Gl ' C0 < Ami ' a)dudv ( 37) 

-\\[{G\G,+G;G 2 )iudv 

~\\\r {G* 2 G r exp(-4mu'a) + G*G 2 .cxp(4mu'a)}diddv 
Applying Equation (28) to Equation (37) results in the following: 
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t(a,0) = s{X,Y) 

2 

>dudv 



- UK ■ |G '' 2 



2 

-\\[{GlG^G;G 2 )ludv . (38) 

+ \ II l Gl I' cos ( 47r t w ^ + vY})dudv 

~ ~ J I G i * ^PC-^M^ + v *1) + & X G 2 .exp(4m[uX + vr])jrfwrfv 



Because the blurred images gj and g2 are both real functions, the following 
properties exist: 

G*(u i v) = G n (-u-v) 

\ G n(~ U - V ) = \ G n( U > V ] 

The full (or at least symmetrical) limits in frequency space allows the last two 
terms of Equation (38) to be joined. 



10 - 1 £ £ {g; (u 9 v)G x (k, v) + G; (u 9 v)G 2 (u, v)\ludv . (40) 



+ \ £ £i Gl < w? v ^| 2 ex p( 4 ^["^ + vY^dudv 



Only the last two terms of Equation (40) are functions of X and Y. These are 
therefore the only terms determining where the minimum of the deviation function s (X, Y) 
is located. In particular, the last two terms can be recognised as having the form of auto 
15 and cross-correlation functions). 
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\ £ fl l G| (m> v) I 2 ex p( 4;r ["^ + v ypudv 

- £ £ G, (- u,—v)G 1 (m, v). exp(4m[uX + vY]jdudv . (41) 

= - f r G \ (~ " - v M G i (- "- v ) - 2G 2 ("> v )}- exp{4m[uX + vY^dudv 



Equation (41) represents the convolution of gvf-x.-.y,) with the difference of 
(gi-x,-y) and 2g2(x,y) evaluated at coordinates {X', Y') , where 

5 

X' = 2X] 

Y' = 2Y | (42) 



Case (c) is special case where: 



10 



f(x,y) = S(x,y) 
b l (x,y) = g l (x,y) 
b 2 (x,y) = g 2 (x,y) 



(43) 



and is illustrated in Figs. 5A and 5B. An alternative solution is to negate the second term 

in Equation (41), and find the maximum instead. This special case is equivalent to 

finding the maximum cross-correlation for the blur functions bi and b2 shown in Figs. 2A 
15 and 2B, with the absolute blur functions shifted by a. However, this equal shift of both 

blur functions does not effect the cross-correlation between the blur functions bi and b2. 

Thus, this special case reduces to the conventional shift solution. 

A person skilled in the art would recognise that many variations of the above 

embodiments exist. For example, the blur function 62 can be chosen such that its duration 
20 comprises multiples of the duration of blur function bj, wherein the multiples are greater 

than two which correspond with the embodiments described above. 

By considering embodiments with higher durations for blur function 62 relative 

to the blur function 6y, a sequence of auto-correlations and cross-correlations are 

evaluated with various integer re-scaling factors. However, the overall method is similar 
25 in that a direct calculation of the minimisation is possible using fast correlation 

techniques. 
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In order to illustrate the above, yet another embodiment is described. Figs. 6A 
and 6B illustrate the x' components of the respective blur functions bj and b 2 resulting 
from two uniform exposures centred on the same time, but with the one exposure three 
times the duration of the other. An alternative embodiment with the same start time can 
5 be analysed in a very similar fashion and will not be explicitly derived. 

This particular relationship between the two exposures results in the following 
parameters: 



a 2 = 3a 
x' 0 =0 

y' 0 =o 



(44) 



10 Applying the parameters of Equations (44) to Equations (12) and (13) gives the 

Fourier spectra: 

/ \ cv \ sin(2^w'a) 

G,(w,v) = F(w,v).— — ; (45) 

ltvu a 



sin(67ru f a ) 

G 2 (u,v) = F(u,v).^— r J - + N(u,v) (46) 

07TU a 

15 

Using the following trigonometric expansion: 

sin(6^'a)= [4cos 2 (2^w'a)-lJsin(2^w'a)= [2cos(4^w'a)+l]sin(2^w'a). (47) 

20 the following total modulus-squared spectral "deviation" t y integrated over a 
representative region of the spatial frequency, must be minimised: 

t(a)= JJJaM>,v)| 2 c/w</v= J^|G 2 (w,v)-G 1 (w,v).cos(2^'a)| 2 ^v. (48) 

25 Expanding Equation (48) gives: 
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t(a) = 



'udv 
(49) 



Thus, by examining each of the terms above, the blur parameter can be directly 
5 estimated, in a manner similar to the previous embodiments. Equation (49) includes two 
cross-correlations re-scaled by factor two and two auto-correlations re-scaled by factors 
four and two respectively. 

When blur functions are considered with the duration of one of the blur functions 
a larger integer value of that of the other, the number of re-scaled auto- and cross- 
10 correlations to be evaluated also increase, but otherwise similar relationships exist. 

Non-uniform exposures can similarly be used. Figs. 7 A and 7B illustrate yet 
another embodiment wherein the x' components of the respective blur functions bj and 
&2 resulting from two non-uniform exposures centred on the same time, but with the one 
exposure twice the duration of the other. An alternative embodiment with the same start 
15 time can be analysed in a very similar fashion and will therefore not be explicitly derived. 

The typical non-uniform exposure pattern is derived from just one convolution. 
The method is readily extended to multiple convolutions. 

The Fourier spectra in this embodiment is as follows: 



20 



G l (u 9 v) = F(u 9 v). 



sm(27m'a) 
2m' a 



(50) 



G 2 («,v) = F(a,v). 



sin(4;zw'a ) 
Am' a 



+ N(u,v) 



(51) 



wherein the integer value m represents the number of times a uniform exposure pattern 
25 has to be convolved to produce the required non-uniform exposure pattern. The 
embodiment illustrated in Figs. 7 A and &B was derived using m~2. 

Minimisation needs to be applied to the following total modulus-squared spectral 
"deviation" 
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t(a)= j^\N(u,v)\ 2 dudv= j ^\G 2 (u 9 v)-G } (u,v).cos m (2m'a)\ 2 dudv . (52) 

Using similar argument to those in previous embodiments, Equation (52) will, in 
general, expand into a series of auto- and cross-correlation with various integer re-scaling 
5 factors: 

t( a )= J^jG 2 | 2 +|G 1 | 2 cos 2m (2^w^)}/^v 
- JJ^G, +G;G 2 )cos m (2;rw'a>/wc/v 

Using m=2 gives: 



(53) 



/ s rr i^i 2 \G.\ 2 \cos(&mi'a)+4cos(4mi'a)+3]/ , , 

" fl >-JIf"l +l ' ll /* j** (54) 

- J^feG, +G;G 2 Jcos(4^w^)+l]/2^wc/v 



Specifically in this the case with m=2, from Equation 54, the cross-correlation is 
re-scaled by a factor of two, and the auto-correlation re-scaled by factors of two and four. 

The foregoing provide direct computational methods for estimating blur 

15 parameters over an infinite plane. For general implementation, the algorithms are 
desirably modified for use with finite and discrete data-sets typical of digitally sample 
images. The computation may be modified to work over regions, typically rectangular 
blocks of pixels, which are significantly larger than the blur parameter sought to be 
estimated, yet small enough so that the blur parameter remains substantially constant over 

20 the region. Also, edge effects which are not present in the infinite plane, may be 
mitigated by suitable image padding and windowing methods. These constraints are 
typical of other correlation-based image analysis techniques and consequently need not be 
discussed further herein. 

Fig. 9 illustrates a block diagram of apparatus 9 for estimating the blur parameters of 

25 blurred images according to an embodiment of the present invention. The apparatus 9 has 
an active image sensor 10, preferably formed in CMOS technology, for capturing the 
blurred images g } and g 2 , a plurality of correlators 20 and 30 for performing 
autocorrelation of the blurred image g\ and cross-correlation between the two images g/ 
and gi respectively, an error function calculator 40 for evaluating an error function t over 
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all possible displacements using the results from the correlators 20 and 30, and an 
extreme locater 50 for finding the displacement with the minimum value for the error 
function t. 

The operation of the image sensor 10, as implemented with CMOS technology, will 

5 be further explained with references to Figs. 10 and 1 1 . Fig. 10 shows a circuit layout of 
a part 12 of the image sensor 10, the part 12 being arranged to capture a single pixel of 
the digital image signal f(x,y) by means of a pixel cell 60. 

The pixel cell 60 is formed of a photodiode 63 which acts as an image pixel sensor 
and which connects to a transistor 64 to discharge a parasitic capacitor Cp according to 

10 the intensity of illumination falling upon the photodiode 63. The parasitic capacitor Cp 
may be formed by a floating diffusion. A transistor switch 62, operated by a reset signal 
'res', acts to reset the capacitor Cp by applying a supply rail potential to the capacitor Cp. 
A signal "TX" is applied to the parasitic capacitor Cp enabling the capacitor Cp to 
discharge through the operation of the photodiode 63. It will be appreciated therefore that 

15 the sensor 10 operates according to negative logic principles where the charge on the 
capacitor Cp is the inverse of the light intensity-duration function applied to the 
photodiode 63. A select signal 'sel' operates a transistor switch 65a which, together with 
a source-follower transistor 65b, couples the voltage on the parasitic capacitor Cp to a 
pixel data bus 61. The duration that the signal TX' is applied is usually called the 

20 exposure time, and the charge of the parasitic capacitor Cp is the value of the pixel cell 
60. A number of the pixel cells 60 may be connected to the pixel data bus 61 . 

In order to provide a sequence of output pixel values of an image, a switching 
arrangement 66 is provided to couple such values to an output 67 of the part 12. The 
sequence may, for example, be a raster sequence. 

25 Signals "Nl" and "N2" are applied to transistor switches SW1 and SW3, causing 

them to conduct thereby providing for noise charge values to be stored on capacitors CI 
and C3 respectively. Such noise values are representative of circuit noise and are 
available on the pixel data bus 61 when the 'sel' signal of each of the cells 60 connected 
to the bus 61 are all disabled or not selected. 

30 Signals 'SI' and 'S2' are applied to turn ON switches SW0 or SW2 respectively, so 

that capacitors CO and C2 are selectively charged according to the pixel value as present 
on the pixel bus 61. 

Signals 'HSTO'and 'HST1" are applied to turn ON switch pairs SW4 and SW5, and 
SW6 and SW7 respectively. The switches SW5 and SW7 act to couple the noise values 
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on the capacitors CI and C3 respectively to a buffer 'bufh', while the pixel values present 
on the capacitors CO and C2 are coupled via operation of the switches SW4 and SW6 
respectively to a buffer 'bufs'. A differential amplifier 'diffamp', connected to the 
buffers 'bufh' and 'bufs', finds the difference between the two signals to output a signal 
5 indicative of the light fallen upon the photodiode 63. 

By forming the image sensor 10 as a matrix of pixel cells 60 formed in rows (lines) 
and columns, overlapped double sampling of an image that gives rise to blur functions 
illustrated in Figs. 4 A and 4B, can be achieved using the circuit 12 of Fig. 10 by applying 
to the circuit 12 a signal sequence as shown in Fig. 11. 

10 When a signal 'selO' is asserted, all the pixel cells 60 in row 0 of the matrix are 

enabled, and each drive a corresponding pixel data bus 61 in the corresponding column. 

The signal 'res' is then asserted, causing all the pixel cells 60 in line 0 to be reset, 
and leaving each cell 60 driving a noise value on the corresponding pixel data bus 61 . 

As the signal 'res' id de-asserted, the signals 'Nl' and 'N2' are then asserted to latch 

15 the noise values into capacitors CI and C3. The signal 'TX' is then asserted for a period 
'Tex'. During the period 'Tex', all the pixel cells 60 in line 0 discharge their 
corresponding parasitic capacitors Cp according to the light intensity on line 0. The 
signal 'SI' is asserted in the middle of the exposure time to store the pixel values at that 
time in the respective capacitors CO. At the end of the exposure time 'Tex', the pixel 

20 values at that time are stored in capacitors C2 by asserting signal 'S2\ The signal 'HST0' 
is then asserted to switch on SW4 and SW5 to find the true values of the pixels during the 
first half of exposure time 'Tex'. The signal 'HST1' is asserted to switch on SW6 and 
SW7 to find the true values of the pixels during the second half of exposure time 'Tex'. 
The signals 'HST0' and 'HST1' are asserted progressively along line 0 for each pixel to 

25 output a sequence of pixel values for the line. This is repeated until all the pixels in line 0 
are read. 

Such a mode of operation results in a pixel sequence as follows, for n pixels in the 

line: 

Pixel_l_half, Pixel_l_foll, Pixel_2_half, Pixel_2_full, . . . Pixel_n_full. 
30 A next line (line 1) is selected by asserting 'sell ', and the above is repeated. 

Alternatively, the mode of operation may be altered to provide the following 
sequence: 

Pixel_l_half, Pixel-2_half ... Pixel_n_half, Pixel l_full, Pixel 2_full, ... 
Pixel n foil. 
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It will be apparent from the forgoing that where SI is activated at the middle of TX 
and S2 at the end, the first image obtained by virtue of SI, for a stationary target which 
presents no blur, will have half the intensity of the second image obtained at S2. As such, 
any image derived at SI will include, for all intents and purposes, the same components 
5 of the image obtained at S2. This is to be contrasted with prior art arrangements where 
single frames or fields are captured one after the other in which case the entire sensor 
reset and newly illuminated. 

Once the blur parameters have been estimated from the two sub-frame images, a 
new, de-blurred image may be generated. This will not be required for typical video 

10 image sequences, because the sub-frame blur will be much smaller than the inter-frame 
movement blur. However, if a still image with minimal motion blur is required, then 
various de-convolution methods may be used to generate such an image. Such methods 
can utilise the estimated blur parameters to obtain a direct solution. Alternatively, 
iterative methods may be used with a first iteration using the estimated blur parameters to 

15 accelerate convergence. 

By utilizing sub-frame or sub-field sampling of an image through the arrangement of 
Fig. 10, it is thus possible to obtain a motion vector from the equivalent of single prior art 
image sample, whereas according to prior art configurations, at least two traditional 
samples would be required. Thus the described embodiment provides for an estimation of 

20 motion blur for each traditional image capture, be it a frame or an interlaced field. 

The above can be readily seen from Figs. 12A to 12E. Fig. 12A shows a prior art 
video signal 200 where an image frame is captured during period 202 and output from a 
prior art sensor during a period 204 prior to the next frame 202 being captured. Fig. 10B 
shows a similar signal 206 for an interlaced frame where a first field 208 is captured and 

25 output over a period 210. A second field 212 is captured and output over a period 214 
prior to the first field of the next frame being captured. 

According to an embodiment of the invention, as shown in Fig. 12C, a first image 
222 is captured immediately followed by a second image 224, the second image 224 for 
example corresponding to that captured during the first field 208 of Fig. 12B. The data of 

30 the two images 222 and 226 is output during a period 226 before images 228 and 230 are 
captured for the next field and output during the period 220. It follows from the above, 
that the field period of the capture device will be defined at least by a time taken to read 
out information of twice number of total pixels of an image sensor of the capture device, 
so that for each field period, at least one estimate of motion is made. 
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Whilst Fig. 12C shows the motion blur images being captured one immediately after 
the other, such need not always be the case. As seen in Fig. 12D, a capture sequence 234 
includes the capture of a first blur image 236 overlapping with the capture of a second 
blur image 238. In Fig. 12E, a capture sequence 240 involves capturing one image 242 
5 over a period twice that of another image 244. 

The blur image processing methods described above and schematically 
illustrated in Fig. 9 may be practiced using a conventional general-purpose computer 
system 100, such as that shown in Fig. 8 wherein the Fourier Transform as well as the 
auto- correlation and cross-correlation calculations, error function calculation and 

10 extreme location may be implemented as software, such as an application program 
executing within the computer system 100. In particular, those calculations are effected 
by instructions in the software that are carried out by the computer. The software may be 
stored in a computer readable medium, including the storage devices described below. 
The software is loaded into the computer from the computer readable medium, and then 

15 executed by the computer. A computer readable medium having such software or 
computer program recorded on it is a computer program product. 

The computer system 100 comprises a computer module 102, input devices such 
as a keyboard 110 and mouse 112, output devices including a display device 104. The 
computer module 102 typically includes at least one processor unit 114, a memory 

20 unit 118, for example formed from semiconductor random access memory (RAM) and 
read only memory (ROM), input/output (I/O) interfaces including a video interface 122, 
and an I/O interface 116 for the keyboard 110 and mouse 112 and peripherals interface 
132, to which an appropriate image sensor 134 configured according to Figs. 10 and 1 1 is 
connected to provide image pixel data to the computer module 102. A storage device 124 

25 is provided and typically includes a hard disk drive 126 and a floppy disk drive 128. A 
magnetic tape drive (not illustrated) may also be used. A CD-ROM drive 120 is typically 
provided as a non- volatile source of data. The components 114 to 128 of the computer 
module 102, typically communicate via an interconnected bus 130 and in a manner which 
results in a conventional mode of operation of the computer system 100 known to those in 

30 the relevant art. Examples of computers on which the embodiments can be practised 
include IBM-PC's and compatibles, Sun Sparcstations or alike computer systems evolved 
therefrom. 

Typically, the application program of the preferred embodiment is resident on 
the hard disk drive 126 and read and controlled in its execution by the processor 114. 
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Intermediate storage of the program may be accomplished using the semiconductor 
memory 118, possibly in concert with the hard disk drive 128. In some instances, the 
application program may be supplied to the user encoded on a CD-ROM or floppy disk 
and read via the corresponding drive 120 or 128. Still further, the software can also be 
5 loaded into the computer system 100 from other computer readable medium including 
magnetic tape, a ROM or integrated circuit, a magneto-optical disk, a radio or infra-red 
transmission channel between the computer module 102 and another device, a computer 
readable card such as a PCMCIA card, and the Internet and Intranets including email 
transmissions and information recorded on websites and the like. The foregoing is merely 

10 exemplary of relevant computer readable mediums. Other computer readable mediums 
may be practiced without departing from the scope and spirit of the invention. 

The calculation of the Fourier transforms as well as the auto-correlation and the 
cross-correlation may alternatively be implemented in dedicated hardware such as one or 
more integrated circuits performing these functions or parts thereof. Such dedicated 

15 hardware may include graphic processors, digital signal processors, or one or more 
microprocessors and associated memories. 

The embodiments described permit the estimation of motion of an object 
captured in an image through obtaining at least two blur images where traditionally only a 
single image would be obtained. The blur images are then processed to determine motion 

20 parameters across the pair of images. The motion parameters may then be associated with 
one of the images, usually the second of the two, two provide an image with 
corresponding motion parameters obtained during the generation of the (second) image. 
The motion parameters may then be used as desired for object identification within the 
image, for example. Where the images for a sequence of such images, the motion 

25 parameters permit the tracking or other monitoring of moving objects contained within 
the image sequence. Motion detection using an embodiment of the present invention is 
desirable because such associates one motion vector map to each frame or field, which 
not only is smaller in magnitude thereby affording finer resolution, but also a more 
accurate representation of motion. 

30 The foregoing describes only some embodiments of the present invention, and 

modifications and/or changes can be made thereto without departing from the scope and 
spirit of the invention, the embodiments being illustrative and not restrictive. 

In the context of this specification, the word "comprising" means "including 
principally but not necessarily solely" or "having" or "including" and not "consisting only 
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of \ Variations of the word comprising, such as "comprise" and "comprises" have 
corresponding meanings. 
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The claims defining the invention are as follows: 

1 . A method for estimating motion of a moving object, said method comprising the 
steps of: 

capturing at least first and second blurred images of said moving object using a 
capture device having a predetermined field rate, the blur in said images arising from at 
least motion blur of said object, said images being captured within a window period, the 
window period being shorter than a period defined by a field rate of the capture device; 

generating an error function, said error function being a function of said first 
blurred image and said second blurred image; 
minimising said error function; and 

estimating said motion of said object from said minimised error function. 

2. A method according to claim 1, wherein said images are captured consecutively, 
15 one immediately after the other, within said window period. 

3. A method according to claim 1, wherein said images are captured sequentially 
within said window period with a time difference between a start time of capture of said 
first blurred image and a start time of capture of said second blurred image. 

20 

4. A method according to claim 1, wherein a start time of capture of the first 
blurred image and a start time of capture of the second blurred image are concurrent. 

5. A method according to claim 2 or 3, wherein said first blurred image has an 
25 exposure duration substantially equal to an exposure duration of the second blurred 

image. 

6. A method according to claim 3 or 4, wherein said second blurred image has an 
exposure duration that is a predetermined integer multiple of an exposure duration of said 

30 first blurred image. 

7. A method according to claim 3 or 4, wherein said exposure durations of said first 
and second blurred images overlap. 
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8. A method according to any one of the preceding claims, wherein said field rate 
of the capture device is defined at least by a time taken to read out information of half the 
number of total pixels of an image sensor of the capture device, so that for each 
corresponding field period, at least one estimate of motion is made. 

5 

9. A method according to any one of the claims 1 to 7, wherein said field rate of the 
capture device is a rate at which plural frames of a video sequence are captured (ie. the 
frame rate). 

10 10. A method according to any one of claims 1 to 7, wherein an exposure pattern 
(profile) of said exposure duration of at least one of said blurred images is non-uniform. 

11. A method according to claim 10, wherein said exposure pattern (profile) 
comprises a triangular profile. 

15 

12. A method for estimating motion of a moving object, said method comprising the 
steps of: 

capturing at least first and second blurred images of said moving object using a 
capture device having a predetermined field rate, the blur in said images arising from at 
20 least motion blur of said object, said images being captured within a window period, the 
window period being shorter than a period defined by a field rate of the capture device; 

generating an error function, said error function comprising a cross-correlation 
term being a cross-correlation between said first blurred image and said second blurred 
image; 

25 minimising said generated error function; and 

estimating said object motion from said minimised error function. 

13. A method according to claim 10, wherein said error function further comprises 
an auto-correlation term being an auto-correlation of said first blurred image. 

30 

14. Apparatus configured to perform the method of any one of the preceding claims. 

15. A motion estimation system comprising: 
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a capture device for capturing at least first and second blurred images of a 
moving object, the blur in said images arising from at least motion blur of said object, 
said images being captured within a window period, the window period being shorter than 
a period defined by a field rate of the capture device; 
5 means for generating an error function, said error function being a function of 

said first blurred image and said second blurred image; 

means for minimising said error function; and 

means for estimating said motion of said object from said minimised error 

function. 

10 

1 6. A motion estimation system comprising: 

a capture device for capturing at least first and second blurred images of a 
moving object, the blur in said images arising from at least motion blur of said object, 
said images being captured within a window period, the window period being shorter than 
15 a period defined by a field rate of the capture device; 

means for generating an error function, said error function comprising a cross- 
correlation term being a cross-correlation between said first blurred image and said 
second blurred image; 

means for minimising said generated error function; and 
20 means for estimating said object motion from said minimised error function. 

17. A system according to claim 15 or 16 wherein said means for generating, said 
means for minimising and said means for estimation collectively comprise a computer 
system incorporating a sequence of program instructions for estimating said motion using 

25 said images output from said capture device. 

18. A method of estimating motion of an object substantially as described herein 
with reference to any one of the embodiments as that embodiment is illustrated in the 
drawings. 

30 

19. Apparatus for performing the method of claim 18. 

20. A computer readable medium comprising a computer program product for 
performing the method of claim 18. 
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